Image analysis method, apparatus, non-transitory computer readable medium, and deep learning algorithm generation method

ABSTRACT

Disclosed is an image analysis method including inputting analysis data, including information regarding an analysis target cell to a deep learning algorithm having a neural network structure, and analyzing an image by calculating, by use of the deep learning algorithm, a probability that the analysis target cell belongs to each of morphology classifications of a plurality of cells belonging to a predetermined cell group.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/406,523, filed on May 8, 2019, titled “Image Analysis Method,Apparatus, Non-Transitory Computer Readable Medium, And Deep LearningAlgorithm Generation Method,” which claims priority to Japanese PatentApplication No. 2018-091776, filed on May 10, 2018, the contents of eachof which are incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to an image analysis method, apparatus,non-transitory computer readable medium, and deep learning algorithmgeneration method which analyze cell morphology.

2. Description of the Related Art

Japanese Translation of PCT International Application Publication No.2016-534709 discloses a cell identification system for processingmicroscopic images. In the cell identification system, a model obtainedthrough training using a machine training technique associates pixels inan obtained image with one or more of cell, cell edge, background, andequivalents. The machine training technique uses a Random ForestDecision Tree technique.

SUMMARY OF THE INVENTION

In cell examination, usually, an examiner observes cells throughmicroscopic observation, and morphologically identifies the types orfeatures of cells. However, cells of the same lineage have similarmorphologies and thus, in order to become able to morphologicallyidentify cells, it is necessary to improve the identification skill byobserving a large number of cell preparations. In particular,identification of abnormal cells which emerge when a person has adisease requires experience. For example, when the emergence frequencyof abnormal cells is low as in the case of myelodysplastic syndromes inan early stage, there is also a risk that an examiner havinginsufficient skills does not notice abnormal cells.

In addition, the number of preparations that an examiner can observe perday is limited, and observing 100 preparations or more per day isburdensome for the examiner.

Increasing the number of cell examination can be achieved by a flow-typeautomatic hemocyte classification apparatus or the like. However,information that can be obtained from such a flow-type automatichemocyte classification apparatus is limited, and it has been difficultto identify hemocytes having low emergence frequencies, such as blast,promyelocyte, and giant platelet.

A method for identifying cells using a machine training technique (alsoreferred to as machine learning) is also known, such as the methoddescribed in Japanese Translation of PCT International ApplicationPublication No. 2016-534709. However, this method requires the user tocreate training data for training a machine learning model, andgeneration of the model requires tremendous labor. Since the usercreates the training data, the number of pieces of training data thatcan be created is limited, and, at present, there are problems in theanalysis accuracy by the machine learning model and the generalizationcapability.

The method described in Japanese Translation of PCT InternationalApplication Publication No. 2016-534709 is a method for identifying acell portion and a non-cell portion in a microscopic image. Therefore,the method cannot identify what type each cell is, what abnormal findingthe cell has, and the like.

The scope of the present invention is defined solely by the appendedclaims, and is not affected to any degree by the statements within thissummary.

The present disclosure is to provide an image analysis method for moreaccurately identifying the morphology of each of a plurality of cellsincluded in an analysis image.

An embodiment of the present disclosure relates to an image analysismethod for analyzing a morphology of a cell by use of a deep learningalgorithm (50, 51) having a neural network structure. In the imageanalysis method, analysis data (80) being generated from an image of ananalysis target cell and including information regarding the analysistarget cell is inputted to a deep learning algorithm (60, 61) having aneural network structure, and a probability that the analysis targetcell belongs to each of morphology classifications of a plurality ofcells belonging to a predetermined cell group is calculated by use ofthe deep learning algorithm. According to the present embodiment,without an examiner performing microscopic observation, it is possibleto obtain the probability that the analysis target cell belongs to eachof the morphology classifications of the plurality of cells belonging tothe predetermined cell group.

Preferably, the image analysis method includes identifying, on the basisof the calculated probability, the morphology classification of theanalysis target cell. According to the present embodiment, without theexaminer performing microscopic observation, it is possible to identifywhich of the morphology classifications corresponds to the analysistarget cell.

Preferably, the predetermined cell group is a group of blood cells.According to the present embodiment, without the examiner performingmicroscopic observation, it is possible to perform morphologyclassification of hemocytes.

Preferably, the predetermined cell group is a group of cells belongingto a predetermined cell lineage. More preferably, the predetermined celllineage is hematopoietic system. According to the present embodiment,without the examiner performing microscopic observation, it is possibleto perform morphology classification of cells belonging to the same celllineage.

Preferably, each morphology classification indicates a type of cell.More preferably, the morphology classifications include: neutrophil,including segmented neutrophil and band neutrophil; metamyelocyte; bonemarrow cell; promyelocyte; blast; lymphocyte; plasma cell; atypicallymphocyte; monocyte; eosinophil; basophil; erythroblast; giantplatelet; platelet aggregate; and megakaryocyte. According to thepresent embodiment, even cells of the same lineage that have similarmorphologies can be identified.

Preferably, each morphology classification indicates an abnormal findingof cell. More preferably, the morphology classifications include atleast one selected from the group consisting of morphological nucleusabnormality, presence of vacuole, granule morphological abnormality,granule distribution abnormality, presence of abnormal granule, cellsize abnormality, presence of inclusion body, and bare nucleus.According to the present embodiment, even a cell exhibiting an abnormalfinding can be identified.

In the embodiment, data regarding the morphology of the cell is dataregarding the type of the cell according to morphologicalclassification, and data regarding a feature of the cell according tomorphological classification. According to this embodiment, amorphological cell type and a morphological cell feature can beoutputted.

In the embodiment, preferably, the deep learning algorithm includes afirst algorithm configured to calculate a probability that the analysistarget cell belongs to each of first morphology classifications of aplurality of cells belonging to a predetermined cell group, and a secondalgorithm configured to calculate a probability that the analysis targetcell belongs to each of second morphology classifications of a pluralityof cells belonging to a predetermined cell group. For example, eachfirst morphology classification is a type of the analysis target cell,and each second morphology classification is an abnormal finding of theanalysis target cell. Accordingly, the identification accuracy of cellshaving similar morphologies can be more improved.

In the embodiment, the analysis data (80) is generated from an image inwhich a blood cell having been subjected to staining is captured. Morepreferably, the staining is selected from Wright's staining, Giemsastaining, Wright-Giemsa staining, and May-Giemsa staining. Accordingly,identification similar to conventional observation under a microscopiccan be performed.

The analysis data (80) and training data (75) include informationregarding brightness of an analysis target image and a training image,and information regarding at least two types of hue thereof.Accordingly, the identification accuracy can be improved.

Another embodiment of the present disclosure relates to an imageanalysis apparatus (200) configured to analyze morphology of a cell byuse of a deep learning algorithm having a neural network structure. Theimage analysis apparatus (200) includes a processing unit (10) by whichanalysis data (80) being generated from an image of an analysis targetcell and including information regarding the analysis target cell isinput into the deep learning algorithm (60, 61) and a probability thatthe analysis target cell belongs to each of morphology classificationsof a plurality of cells belonging to a predetermined cell group iscalculated by use of the deep learning algorithm (60, 61). Preferably,each morphology classification indicates a type of cell. Preferably,each morphology classification indicates an abnormal finding of cell.

Another embodiment of the present disclosure relates to a non-transitorycomputer readable medium storing programs executable by a processor toperform image analysis for analyzing cell morphology by use of a deeplearning algorithm (60, 61) having a neural network structure. Theprograms cause a processor to execute a process in which analysis data(83) being generated from an image of an analysis target cell andincluding information regarding the analysis target cell is input intothe deep learning algorithm, and a probability that the analysis targetcell belongs to each of morphology classifications of a plurality ofcells belonging to a predetermined cell group is calculated by use ofthe deep learning algorithm (60, 61). Preferably, each morphologyclassification indicates a type of cell. Preferably, each morphologyclassification indicates an abnormal finding of cell.

Another embodiment of the present disclosure relates to a method forgenerating a trained deep learning algorithm (60, 61). In the presentembodiment, training data including information regarding a cell isinputted into an input layer (50 a, 50 b) of a neural network (50, 51),and a label value associated with each of morphology classifications ofa plurality of cells belonging to a predetermined cell group is inputtedas an output layer (51 a, 51 b). Preferably, each morphologyclassification indicates a type of cell. Preferably, each morphologyclassification indicates an abnormal finding of cell.

By use of the image analysis apparatus (200) and the trained deeplearning algorithm (60, 61), it is possible to identify themorphological cell type and cell feature, without being affected by theskill of an examiner.

The morphology of each of a plurality of cells included in an analysisimage can be identified. As a result, cell examination not affected bythe skill of an examiner can be performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing the outline of the present disclosure;

FIG. 2 is a schematic diagram showing an example of a procedure forgenerating training data, and a procedure for training a first deeplearning algorithm and a second deep learning algorithm;

FIG. 3 shows an example of label values;

FIG. 4 is a schematic diagram showing an example of a procedure forgenerating analysis data and a procedure for identifying a cell using adeep learning algorithm;

FIG. 5 is a schematic diagram showing a configuration example of animage analysis system 1;

FIG. 6 is a block diagram showing an example of a hardware configurationof a vendor-side apparatus 100;

FIG. 7 is a block diagram showing an example of a hardware configurationof a user-side apparatus 200;

FIG. 8 is a block diagram for describing an example of the function of adeep learning apparatus 100A;

FIG. 9 is a flow chart showing an example of the flow of a deep learningprocess;

FIG. 10A is a schematic diagram for describing a neural network;

FIG. 10B is a schematic diagram for describing the neural network;

FIG. 10C is a schematic diagram for describing the neural network;

FIG. 11 is a block diagram for describing an example of the function ofan image analysis apparatus 200A;

FIG. 12 is a flow chart showing an example of the flow of an imageanalysis process;

FIG. 13 is a schematic diagram of a configuration example of an imageanalysis system 2;

FIG. 14 is a block diagram for describing an example of the function ofan integrated-type image analysis apparatus 200B;

FIG. 15 is a schematic diagram of a configuration example of an imageanalysis system 3;

FIG. 16 is a block diagram for describing an example of the function ofan integrated-type image analysis apparatus 100B;

FIG. 17 shows an identification result of the type of cell using a deeplearning algorithm; and

FIG. 18 shows an identification result of the feature of cell using thedeep learning algorithm.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, the outline and embodiments of the present disclosure willbe described in detail with reference to the attached drawings. In thefollowing description and the drawings, the same reference characterdenotes the same or like component, and description thereof is omitted.

1. Image Analysis Method

A first embodiment of the present disclosure relates to an imageanalysis method for analyzing cell morphology. In the image analysismethod, analysis data including information regarding an analysis targetcell is inputted to a classifier that includes a deep learning algorithmhaving a neural network structure. The classifier calculates theprobability that the analysis target cell belongs to each of morphologyclassifications of a plurality of cells belonging to a predeterminedcell group. Preferably, the image analysis method further includesidentifying, on the basis of the probability, which of the morphologyclassifications of the plurality of cells belonging to the predeterminedcell group corresponds to the analysis target cell.

In the first embodiment, the analysis target cell belongs to apredetermined cell group. The predetermined cell group is a group ofcells that form each organ in the body of a mammal or a bird. Thepredetermined cell group, in a normal state, includes a plurality oftypes of cells morphologically classified through histologicalmicroscopic observation or cytological microscopic observation. Themorphological classification (also referred to as “morphologyclassification”) includes classification of the type of cell andclassification of morphological feature of cell. Preferably, theanalysis target cell is a group of cells that belong to a predeterminedcell lineage that belongs to a predetermined cell group. Thepredetermined cell lineage is a cell group that belongs to the samelineage that has differentiated from one type of tissue stem cell.Preferably, the predetermined cell lineage is the hematopoietic system,and more preferably, cells in blood (also referred to as blood cells).

In a conventional method, a human observes, in a microscopic brightfield, a preparation having been subjected to bright field staining,whereby hematopoietic cells are morphologically classified. Preferably,the staining is selected from Wright's staining, Giemsa staining,Wright-Giemsa staining, and May-Giemsa staining. More preferably, thestaining is May-Giemsa staining. The preparation is not restricted aslong as the preparation allows individual observation of the morphologyof respective cells belonging to a predetermined cell group. Examples ofthe preparation include a smear preparation and an impressionpreparation. Preferably, the preparation is a smear preparation usingperipheral blood or bone marrow as a sample, and more preferably, is asmear preparation of peripheral blood.

In morphological classification, the type of blood cells includesneutrophil, including segmented neutrophil and band neutrophil;metamyelocyte; bone marrow cell; promyelocyte; blast; lymphocyte; plasmacell; atypical lymphocyte; monocyte, eosinophil, basophil, erythroblast(which is nucleated erythrocyte and includes proerythroblast, basophilicerythroblast, polychromatic erythroblast, orthochromatic erythroblast,promegaloblast, basophilic megaloblast, polychromatic megaloblast, andorthochromatic megaloblast); giant platelet; platelet aggregate;megakaryocyte (which is nucleated megakaryocyte and includesmicromegakaryocyte); and the like.

The predetermined cell group may include abnormal cells that exhibitmorphologically abnormal findings, in addition to normal cells.Abnormality appears as a morphologically classified cell feature.Examples of abnormal cells are cells that emerge when a person has apredetermined disease, and are tumor cells, for example. In the case ofthe hematopoietic system, the predetermined disease is a diseaseselected from the group consisting of myelodysplastic syndromes,leukemia (including acute myeloblastic leukemia, acute myeloblasticleukemia, acute promyelocytic leukemia, acute myelomonocytic leukemia,acute monocytic leukemia, erythroleukemia, acute megakaryoblasticleukemia, acute myeloid leukemia, acute lymphoblastic leukemia,lymphoblastic leukemia, chronic myelogenous leukemia, chroniclymphocytic leukemia, and the like), malignant lymphoma (Hodgkin'slymphoma, non-Hodgkin's lymphoma, and the like), and multiple myeloma.In the case of the hematopoietic system, the cell having an abnormalfinding is a cell that has at least one type of morphological featureselected from the group consisting of: morphological nucleusabnormality; presence of vacuole, granule morphological abnormality;granule distribution abnormality; presence of abnormal granule; cellsize abnormality; presence of inclusion body; and bare nucleus.

Examples of the morphological nucleus abnormality include nucleusbecoming small, nucleus becoming large, nucleus becoming hypersegmented,nucleus that should be segmented in a normal state but has not beensegmented (including pseudo-Pelger anomaly and the like), presence ofvacuole, swelled nucleolus, cleaved nucleus, a single cell that shouldhave one nucleus but has the anomaly of having two, and the like.

Examples of abnormality in the morphology of an entire cell includepresence of vacuole in cytoplasm (also referred to as vacuolardegeneration), morphological abnormality in granule (such as azurophilgranule, neturophil granule, eosinophil granule, and basophil granule),presence of abnormality in distribution (excess, decrease, ordisappearance) of the above-mentioned granules, presence of abnormalgranule (for example, toxic granule), cell size abnormality (larger orsmaller than normal cell), presence of inclusion body (Dohle body, Auerbody, and the like), bare nucleus, and the like.

<Outline of Image Analysis Method>

The outline of an image analysis method is described with reference toFIG. 1 .

A classifier used in the image analysis method includes a plurality ofdeep learning algorithms (also simply referred to as “algorithm”) eachhaving a neural network structure. Preferably, the classifier includes afirst deep learning algorithm (50) and a second deep learning algorithm(51). The first deep learning algorithm (50) extracts the featurequantity of a cell, and the second deep learning algorithm (51)identifies the analysis target cell on the basis of the feature quantityextracted by the first deep learning algorithm. More preferably, at thedownstream of the first deep learning algorithm as shown in FIG. 1 , inaddition to the second deep learning algorithm, the classifier mayinclude a plurality of types of deep learning algorithms (which aresometimes numbered as the second, the third, the fourth, the fifth, . .. , the i-th) having been trained in accordance with the objective ofthe identification. For example, the second deep learning algorithmidentifies the type of cell based on the morphological classificationdescribed above. For example, the third deep learning algorithmidentifies the feature of cell, for each feature, based on themorphological classification described above. Preferably, the first deeplearning algorithm is a convolution connect neural network, and thesecond deep learning algorithm and thereafter at the downstream of thefirst deep learning algorithm are each a full connect neural network.

Next, a method for generating training data 75 and an image analysismethod are described with reference to the examples shown in FIG. 2 toFIG. 4 . In the following, for convenience of description, the firstdeep learning algorithm and the second deep learning algorithm are used.

<Generation of Training Data>

A training image 70 that is used for training a deep learning algorithmis a captured image of a cell whose type of cell (also referred to ascell type) and feature of cell (also referred to as cell feature) basedon morphological classification that corresponds to the analysis targetcell are known. Preferably, the preparation used for capturing thetraining image 70 is created from a sample that contains the same typeof cells as the analysis target cell, by a preparation creation methodand a staining method similar to those for a preparation that includesthe analysis target cell. Preferably, the training image 70 is capturedin a condition similar to the image capturing condition for the analysistarget cell.

The training image 70 can be obtained in advance for each cell by useof, for example, a known light microscope or an imaging apparatus suchas a virtual slide scanner. In the example shown in FIG. 2 , thetraining image 70 is obtained by reducing a raw image captured in 360pixels×365 pixels by Sysmex DI-60 into 255 pixels×255 pixels. However,this reduction is not mandatory. The number of pixels of the trainingimage 70 is not restricted as long as analysis can be performed, but thenumber of pixels of one side thereof is preferably greater than 100. Inthe example shown in FIG. 2 , erythrocytes are present around theneutrophil, but the image may be trimmed such that only the target cellis included in the image. If, at least, one cell, for which training isto be performed (erythrocytes, and platelets of normal sizes may beincluded), is included in one image and the pixels corresponding to thecell, for which training is to be performed, exist by about 1/9 of thetotal pixels of the image, the image can be used as the training image70.

For example, in the present embodiment, preferably, image capturing bythe imaging apparatus is performed in RGB colors, CMY colors, or thelike. Preferably, as for a color image, the darkness/paleness orbrightness of each of primary colors, such as red, green, and blue, orcyan, magenta, and yellow, is expressed by a 24 bit value (8 bits×3colors). It is sufficient that the training image 70 includes at leastone hue, and the darkness/paleness or brightness of the hue, but morepreferably, includes at least two hues and the darkness/paleness orbrightness of each hue. Information including hue and thedarkness/paleness or brightness of the hue is also called tone.

Next, information of tone of each pixel is converted from, for example,RGB colors into a format that includes information of brightness andinformation of hue. Examples of the format that includes information ofbrightness and information of hue include YUV (YCbCr, YPbPr, YIQ, andthe like). Here, an example of converting to a YCbCr format isdescribed. Since the training image is in RGB colors, conversion intobrightness 72Y, first hue (for example, bluish color) 72Cb, and secondhue (for example, reddish color) 72Cr is performed. Conversion from RGBto YCbCr can be performed by a known method. For example, conversionfrom RGB to YCbCr can be performed according to International StandardITU-R BT.601. The brightness 72Y, the first hue 72Cb, and the second hue72Cr after the conversion can be each expressed as a matrix of gradationvalues as shown in FIG. 2 (hereinafter, also referred to as tonematrices 72 y, 72 cb, and 72 cr). The brightness 72Y, the first hue72Cb, and the second hue 72Cr are each expressed in 256 gradationsconsisting of 0 to 255 gradations. Here, instead of the brightness 72Y,the first hue 72Cb, and the second hue 72Cr, the training image may beconverted into the three primary colors of red R, green G, and blue B,or the three primary colors of pigment of cyan C, magenta M, and yellowY.

Next, on the basis of the tone matrices 72 y, 72 cb, and 72 cr, for eachpixel, tone vector data 74 is generated by combining three gradationvalues of the brightness 72 y, the first hue 72 cb, and the second hue72 cr.

Next, for example, since the training image 70 in FIG. 2 is of asegmented neutrophil, each tone vector data 74 generated from thetraining image 70 in FIG. 2 is provided with “1” as a label value 77which indicates that the image is of a segmented neutrophil, wherebytraining data 75 is obtained. In FIG. 2 , for convenience, the trainingdata 75 is expressed by 3 pixels×3 pixels. However, in actuality, thetone vector data exists by the number of pixels that have been obtainedat the capture of the image of the training data 70.

FIG. 3 shows an example of the label value 77. As the label value, alabel value 77 that is different according to the type of cell and thepresence/absence of a feature of each cell is provided.

<Outline of Deep Learning>

Using FIG. 2 as an example, the outline of neural network training isdescribed. Preferably, both a first neural network 50 and a secondneural network 51 are convolution neural networks. The number of nodesat an input layer 50 a in the first neural network 50 corresponds to theproduct of the number of pixels of the training data 75 that isinputted, and the number of brightness and hue (for example, in theabove example, three, i.e., the brightness 72 y, the first hue 72 cb,and the second hue 72 cr) included in the image. The tone vector data 74is inputted, as a set 76 thereof, to the input layer 50 a of the firstneural network 50. Using the label value 77 of each pixel of thetraining data 75 as an output layer 50 b of the first neural network,the first neural network 50 is trained.

On the basis of the training data 75, the first neural network 50extracts feature quantities with respect to the morphological cell typeor cell feature described above. The output layer 50 b of the firstneural network outputs a result reflecting these feature quantities.Each result outputted from a softmax function of the first neuralnetwork 50 is inputted in an input layer 51 a of the second neuralnetwork 51. Since cells that belong to a predetermined cell lineage havesimilar cell morphologies, a deep learning algorithm 51 having thesecond neural network 51 is further specialized in identification of amorphologically specific cell type or morphologically specific cellfeatures, so that the deep learning algorithm is trained. Therefore, thelabel value 77 of the training data 75 is also inputted to the outputlayer of the second neural network. Reference characters 50 c and 51 cin FIG. 2 represent middle layers.

The first deep learning algorithm 60 having the thus-trained firstneural network 60, and the second deep learning algorithm 61 having thethus-trained second neural network 61 are combined to be used as aclassifier for identifying which of the morphologically classified typesof a plurality of cells belonging to a predetermined cell groupcorresponds to the analysis target cell.

<Image Analysis Method>

FIG. 4 shows an example of an image analysis method. In the imageanalysis method, analysis data 81 is generated from an analysis image 78in which the analysis target cell is captured. The analysis image 78 isan image in which the analysis target cell is captured. The analysisimage 78 can be obtained by use of, for example, a known lightmicroscope or a known imaging apparatus such as a virtual slide scanner.In the example shown in FIG. 4 , the analysis image 78 is obtained byreducing a raw image captured in 360 pixels×365 pixels by Sysmex DI-60into 255 pixels×255 pixels, as in the case of the training image 70.However, this reduction is not mandatory. The number of pixels of theanalysis image 78 is not restricted as long as analysis can beperformed, but the number of pixels of one side thereof is preferablygreater than 100. In the example shown in FIG. 4 , erythrocytes arepresent around the segmented neutrophil, but the image may be trimmedsuch that only the target cell is included in the image. If, at least,one analysis target cell is included in one image (erythrocytes, andplatelets of normal sizes may be included) and the pixels correspondingto the analysis target cell exist by about 1/9 of the total pixels ofthe image, the image can be used as the analysis image 78.

For example, in the present embodiment, preferably, image capturing bythe imaging apparatus is performed in RGB colors, CMY colors, or thelike. Preferably, as for a color image, the darkness/paleness orbrightness of each of primary colors, such as red, green, and blue, orcyan, magenta, and yellow, is expressed by a 24 bit value (8 bits×3colors). It is sufficient that the analysis image 78 includes at leastone hue, and the darkness/paleness or brightness of the hue, but morepreferably, includes at least two hues and the darkness/paleness orbrightness of each hue. Information including hue and thedarkness/paleness or brightness of the hue is also called tone.

For example, the format of RGB colors is converted into a format thatincludes information of brightness and information of hue. Examples ofthe format that includes information of brightness and information ofhue include YUV (YCbCr, YPbPr, YIQ, and the like). Here, an example ofconverting to a YCbCr format is described. Since the analysis image isin RGB colors, conversion into brightness 79Y, first hue (for example,bluish color) 79Cb, and second hue (for example, reddish color) 79Cr isperformed. Conversion from RGB to YCbCr can be performed by a knownmethod. For example, conversion from RGB to YCbCr can be performedaccording to International Standard ITU-R BT.601. The brightness 79Y,the first hue 79Cb, and the second hue 79Cr after the conversion can beeach expressed as a matrix of gradation values as shown in FIG. 2(hereinafter, also referred to as tone matrices 79 y, 79 cb, 79 cr). Thebrightness 72Y, the first hue 72Cb, and the second hue 72Cr are eachexpressed in 256 gradations consisting of 0 to 255 gradations. Here,instead of the brightness 79Y, the first hue 79Cb, and the second hue79Cr, the analysis image may be converted into the three primary colorsof red R, green G, and blue B, or the three primary colors of pigment ofcyan C, magenta M, and yellow Y.

Next, on the basis of the tone matrices 79 y, 79 cb, and 79 cr, for eachpixel, tone vector data 80 is generated by combining three gradationvalues of the brightness 79 y, the first hue 79 cb, and the second hue79 cr. A set of the tone vector data 80 generated from one analysisimage 78 is generated as the analysis data 81.

Preferably, the generation of the analysis data 81 and the generation ofthe training data 75 have, at least, the same image capturing conditionand the same condition of generating, from each image, vector data to beinputted into neural networks.

The analysis data 81 is inputted to an input layer 60 a of the firstneural network 60 forming the first deep learning algorithm 60 havingbeen trained. The first deep learning algorithm extracts featurequantities from the analysis data 81, and outputs the result from anoutput layer 60 b of the first neural network 60. The value outputtedfrom the output layer 60 b is a probability that the analysis targetcell included in the analysis image belongs to each of the morphologicalcell classification or feature inputted as the training data.

Next, the result outputted from the output layer 60 b is inputted to aninput layer 61 a of the second neural network 61 forming the second deeplearning algorithm 61 having been trained. On the basis of the inputtedfeature quantities, the second deep learning algorithm 61 outputs, froman output layer 61 b, a probability that the analysis target cellincluded in the analysis image belongs to each of the morphological cellclassification or feature inputted as the training data. Further, it isdetermined that the analysis target cell included in the analysis imagebelongs to a morphological classification that has the highest value inthe probabilities, and a label value associated with the morphologicalcell type or cell feature is outputted. The label value itself, or dataobtained by replacing the label value with information indicating thepresence/absence of a morphological cell type or cell feature (forexample, a term), is outputted as data 83 regarding the cell morphology.In FIG. 4 , from the analysis data 81, a label value “1” is outputted bythe classifier as a label value 82 having the highest possibility, andcharacter data “segmented neutrophil” corresponding to this label valueis outputted as the data 83 regarding the cell morphology.

Reference characters 60 c and 61 c in FIG. 4 represent middle layers.

2. Image Analysis System 1

<Configuration of Image Analysis System 1>

A second embodiment of the present disclosure relates to an imageanalysis system.

With reference to FIG. 5 , an image analysis system according to thesecond embodiment includes a deep learning apparatus 100A and an imageanalysis apparatus 200A. A vendor-side apparatus 100 operates as thedeep learning apparatus 100A and a user-side apparatus 200 operates asthe image analysis apparatus 200A. The deep learning apparatus 100Acauses the neural network 50 to learn by use of training data, andprovides a user with the deep learning algorithm 60 having been trainedby use of the training data. The deep learning algorithm configured bythe neural network 60 having learned is provided to the image analysisapparatus 200A from the deep learning apparatus 100A through a storagemedium 98 or a network 99. The image analysis apparatus 200A analyzes animage of the analysis target by use of the deep learning algorithmconfigured by the neural network 60 having learned.

The deep learning apparatus 100A is implemented as a general purposecomputer, for example, and performs a deep learning process on the basisof a flow chart described later. The image analysis apparatus 200A isimplemented as a general purpose computer, for example, and performs animage analysis process on the basis of a flow chart described later. Thestorage medium 98 is a computer-readable, non-transitory, and tangiblestorage medium, such as a DVD-ROM, or a USB memory.

The deep learning apparatus 100A is connected to an imaging apparatus300. The imaging apparatus 300 includes an image pickup device 301 and afluorescence microscope 302, and captures a bright field image of alearning preparation 308 set on a stage 309. The training preparation308 has been subjected to the staining described above. The deeplearning apparatus 100A obtains the training image 70 captured by theimaging apparatus 300.

The image analysis apparatus 200A is connected to an imaging apparatus400. The imaging apparatus 400 includes an image pickup device 401 and afluorescence microscope 402, and captures a bright field image of ananalysis target preparation 408 set on a stage 409. The analysis targetpreparation 408 has been stained in advance as described above. Theimage analysis apparatus 200A obtains an analysis target image 78captured by the imaging apparatus 400.

As the imaging apparatus 300, 400, a known light microscope, a knownvirtual slide scanner, or the like that has a function of capturingimages of preparations can be used.

<Hardware Configuration of Deep Learning Apparatus>

With reference to FIG. 6 , the vendor-side apparatus 100 (deep learningapparatus 100A, deep learning apparatus 100B) includes a processing unit10 (10A, 10B), an input unit 16, and an output unit 17.

The processing unit 10 includes a CPU (Central Processing Unit) 11 whichperforms data processing described later, a memory 12 to be used as awork area for data processing, a storage unit 13 which stores therein aprogram and process data described later, a bus 14 which transmits databetween units, an interface unit 15 which inputs/outputs data withrespect to an external apparatus, and a GPU (Graphics Processing Unit)19. The input unit 16 and the output unit 17 are connected to theprocessing unit 10. For example, the input unit 16 is an input devicesuch as a keyboard or a mouse, and the output unit 17 is a displaydevice such as a liquid crystal display. The GPU19 functions as anaccelerator that assists arithmetic processing (for example, parallelarithmetic processing) performed by the CPU 11. That is, the processingperformed by the CPU 11 described below also includes processingperformed by the CPU 11 using the GPU19 as an accelerator.

In order to perform the process of each step described below withreference to FIG. 8 , the processing unit 10 has previously stored, inthe storage unit 13, a program according to the present disclosure andthe neural network 50 before being trained, in an execute form, forexample. The execute form is a form generated as a result of aprogramming language being converted by a compiler, for example. Theprocessing unit 10 uses the program stored in the storage unit 13, toperform a training process for the first neural network 50 and thesecond neural network 51 which are not yet trained.

In the description below, unless otherwise specified, the processperformed by the processing unit 10 means a process performed by the CPU11 on the basis of the program and the neural network 50 stored in thestorage unit 13 or the memory 12. The CPU 11 temporarily storesnecessary data (such as intermediate data being processed) using thememory 12 as a work area, and stores as appropriate, in the storage unit13, data to be saved for a long time such as arithmetic calculationresults.

<Hardware Configuration of Image Analysis Apparatus>

With reference to FIG. 7 , the user-side apparatus 200 (image analysisapparatus 200A, image analysis apparatus 200B, image analysis apparatus200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26,and an output unit 27.

The processing unit 20 includes a CPU (Central Processing Unit) 21 whichperforms data processing described later, a memory 22 to be used as awork area for data processing, a storage unit 23 which stores therein aprogram and process data described later, a bus 24 which transmits databetween units, an interface unit 25 which inputs/outputs data withrespect to an external apparatus, and a GPU (Graphics Processing Unit)29. The input unit 26 and the output unit 27 are connected to theprocessing unit 20. For example, the input unit 26 is an input devicesuch as a keyboard or a mouse, and the output unit 27 is a displaydevice such as a liquid crystal display. The GPU 29 functions as anaccelerator that assists arithmetic processing (for example, parallelarithmetic processing) performed by the CPU 21. That is, the processingperformed by the CPU 21 in the description below also includesprocessing performed by the CPU 21 using the GPU 29 as an accelerator.

In order to perform the process of each step in the image analysisprocess below, the processing unit 20 has previously stored, in thestorage unit 23, a program according to the present disclosure and thedeep learning algorithm 60 of the neural network structure having beentrained, in an execute form, for example. The execute form is a formgenerated as a result of a programming language being converted by acompiler, for example. The processing unit 20 uses the second deeplearning algorithm 61, and the first deep learning algorithm 60 and theprogram stored in the storage unit 23, to perform a process.

In the description below, unless otherwise specified, the processperformed by the processing unit 20 means a process performed by the CPU21 of the processing unit 20 in actuality, on the basis of the programand the deep learning algorithm 60 stored in the storage unit 23 or thememory 22. The CPU 21 temporarily stores necessary data (such asintermediate data being processed) using the memory 22 as a work area,and stores as appropriate, in the storage unit 23, data to be saved fora long time such as arithmetic calculation results.

<Function Block and Processing Procedure>

(Deep Learning Process)

With reference to FIG. 8 , the processing unit 10A of the deep learningapparatus 100A according to the present embodiment includes a trainingdata generation unit 101, a training data input unit 102, and analgorithm update unit 103. These function blocks are realized when aprogram for causing a computer to execute the deep learning process isinstalled in the storage unit 13 or the memory 12 of the processing unit10A, and the program is executed by the CPU 11. A training data database(DB) 104 and an algorithm database (DB) 105 are stored in the storageunit 13 or the memory 12 of the processing unit 10A.

Each training image 70 is captured in advance by the imaging apparatus300 and is stored in advance in the storage unit 13 or the memory 12 ofthe processing unit 10A. The first deep learning algorithm 50 and thesecond deep learning algorithm 51 are stored in advance in the algorithmdatabase 105, in association with the morphological cell type or cellfeature to which the analysis target cell belongs, for example.

The processing unit 10A of the deep learning apparatus 100A performs theprocess shown in FIG. 9 . With reference to the function blocks shown inFIG. 8 , the processes of steps S11, S12, S16, and S17 are performed bythe training data generation unit 101. The process of step S13 isperformed by the training data input unit 102. The processes of stepsS14 and S15 are performed by the algorithm update unit 103.

An example of the deep learning process performed by the processing unit10A is described with reference to FIG. 9 .

First, the processing unit 10A obtains training images 70. Each trainingimage 70 is obtained via the I/F unit 15 through an operation by anoperator, from the imaging apparatus 300, from the storage medium 98, orvia a network. When the training image 70 is obtained, informationregarding which of the morphologically classified cell type and/or themorphological cell feature is indicated by the training image 70 is alsoobtained. The information regarding which of the morphologicallyclassified cell type and/or the morphological cell feature is indicatedmay be associated with the training image 70, or may be inputted by theoperator through the input unit 16.

In step S11, the processing unit 10A converts the obtained trainingimage 70 into brightness Y, first hue Cb, and second hue Cr, andgenerates tone vector data 74 in accordance with the procedure describedin the training data generation method above.

In step S12, the processing unit 10A provides a label value thatcorresponds to the tone vector data 74, on the basis of the informationregarding which of the morphologically classified cell type and/or thecell feature in morphological classification is being indicated, theinformation being associated with the training image 70, and the labelvalue associated with the morphologically classified cell type or thecell feature in morphological classification stored in the memory 12 orthe storage unit 13. In this manner, the processing unit 10A generatesthe training data 75.

In step S13 shown in FIG. 9 , the processing unit 10A trains the firstneural network 50 and the second neural network 51 by use of thetraining data 75. Training results of the first neural network 50 andthe second neural network 51 are accumulated every time training isperformed by use of a plurality of the training data 75.

In the image analysis method according to the present embodiment, theconvolution neural network is used, and the stochastic gradient descentmethod is used. Therefore, in step S14, the processing unit 10Adetermines whether training results for a predetermined number of trialshave been accumulated. When the training results for the predeterminednumber of trials have been accumulated (YES), the processing unit 10Aadvances to the process in step S15, and when the training results forthe predetermined number of trials have not been accumulated (NO), theprocessing unit 10A advances to the process in step S16.

Next, when the training results for the predetermined number of trialshave been accumulated, the processing unit 10A updates, in step S15,connection weights w of the first neural network 50 and the secondneural network 51, by use of the training results accumulated in stepS13. In the image analysis method according to the present embodiment,since the stochastic gradient descent method is used, the connectionweights w of the first neural network 50 and the second neural network51 are updated at a stage where learning results for the predeterminednumber of trials have been accumulated. Specifically, the process ofupdating the connection weights w is a process of performing calculationaccording to the gradient descent method, expressed in Formula 11 andFormula 12 described later.

In step S16, the processing unit 10A determines whether or not the firstneural network 50 and the second neural network 51 have been trained bya prescribed number of training data 75. When training has beenperformed by the prescribed number of training data 75 (YES), the deeplearning process ends.

When the first neural network 50 and the second neural network 51 havenot been trained by the prescribed number of training data 75 (NO), theprocessing unit 10A advances from step S16 to step S17, and performs theprocesses from step S1 to step S16 with respect to the next trainingimage 70.

In accordance with the process described above, the first neural network50 and the second neural network 51 are trained and the first deeplearning algorithm 60 and the second deep learning algorithm 61 areobtained.

(Structure of Neural Network)

As described above, the present embodiment uses the convolution neuralnetwork. FIG. 10A shows an example of the structures of the first neuralnetwork 50 and the second neural network 51. The first neural network 50and the second neural network 51 include: the input layers 50 a, 51 a;the output layers 50 b, 51 b; and the middle layers 50 c, 51 c betweenthe input layers 50 a, 51 a and the output layers 50 b, 51 b. Eachmiddle layer 50 c, 51 c is composed of a plurality of layers. The numberof layers forming the middle layer 50 c, 51 c can be 5 or greater, forexample.

In the first neural network 50 and the second neural network 51, aplurality of nodes 89 arranged in a layered manner are connected betweenlayers. Accordingly, information propagates only in one directionindicated by the arrow D in the figure, from the input side layer 50 a,51 a to the output side layer 50 b, 51 b.

(Calculation at Each Node)

FIG. 10B is a schematic diagram illustrating calculation performed ateach node. The node 89 receives a plurality of inputs and calculates oneoutput (z). In the case of the example shown in FIG. 10B, the node 89receives four inputs. The total input (u) received by the node 89 isexpressed by Formula 1 below.[Math. 1]u=w ₁ x ₁ =w ₂ x ₂ +w ₃ x ₃ +w ₄ x ₄ +b  (Formula 1)

Each input is multiplied by a different weight. In Formula 1, b is avalue called bias. The output (z) of the node serves as an output of apredetermined function f with respect to the total input (u) expressedby Formula 1, and is expressed by Formula 2 below. The function f iscalled an activation function.[Math. 2]z=ƒ(u)  (Formula 2)

FIG. 10C is a schematic diagram illustrating calculation between nodes.In neural network 50, with respect to the total input (u) expressed byFormula 1, nodes that output results (z) each expressed by Formula 2 arearranged in a layered manner. Outputs from the nodes of the previouslayer serve as inputs to the nodes of the next layer. In the exampleshown in FIG. 10C, the outputs from nodes 89 a in the left layer in thefigure serve as inputs to nodes 89 b in the right layer. Each node 89 bin the right layer receives outputs from the respective nodes 89 a inthe left layer. The connection between each node 89 a in the left layerand each node 89 b in the right layer is multiplied by a differentweight. When the respective outputs from the plurality of nodes 89 a ofthe left layer are defined as x₁ to x₄, the inputs to the respectivethree nodes 89 b in the right layer are expressed by Formula 3-1 toFormula 3-3 below.[Math. 3]u ₁ =w ₁₁ x ₁ +w ₁₂ x ₂ +w ₁₃ x ₃ +w ₁₄ x ₄ +b ₁  (Formula 3-1)u ₂ =w ₂₁ x ₁ +w ₂₂ x ₂ +w ₂₃ x ₃ +w ₂₄ x ₄ +b ₂  (Formula 3-2)u ₃ =w ₃₁ x ₁ +w ₃₂ x ₂ +w ₃₃ x ₃ +w ₃₄ x ₄ +b ₃  (Formula 3-3)

When Formula 3-1 to Formula 3-3 are generalized, Formula 3-4 isobtained. Here, i=1, . . . I, and j=1, . . . J.[Math. 4]u _(j)=Σ_(t=1) ^(t) w _(jt) x _(t) +b _(j)  (Formula 3-4)

When Formula 3-4 is applied to an activation function, an output isobtained. The output is expressed by Formula 4 below.[Math.5]z _(j)=ƒ(u _(j)) (j=1,2,3)  (Formula 4)(Activation Function)

In the image analysis method according to the embodiment, a rectifiedlinear unit function is used as the activation function. The rectifiedlinear unit function is expressed by Formula 5 below.[Math. 6]ƒ(u)=max(u,0)  (Formula 5)

Formula 5 is a function obtained by setting u=0 to the part u<0 of thelinear function with z=u. In the example shown in FIG. 10C, usingFormula 5, the output from the node of j=1 is expressed by the formulabelow.z ₁=max((w ₁₁ x ₁ +w ₁₂ x ₂ +w ₁₃ x ₃ +w ₁₄ x ₄ +b ₁),0)  [Math. 7](Learning of Neural Network)

If the function expressed by use of the neural network is defined asy(x:w), the function y(x:w) changes when a parameter w of the neuralnetwork is changed. Adjusting the function y(x:w) such that the neuralnetwork selects a more suitable parameter w with respect to the input xis referred to as learning of the neural network. It is assumed that aplurality of pairs of an input and an output of the function expressedby use of the neural network have been provided. If a desirable outputfor an input x is defined as d, the pairs of the input/output are givenas {(x₁,d₁), (x₂,d₂), . . . , (x_(n),d_(n))}. The set of pairs eachexpressed as (x,d) is referred to as training data. Specifically, theset of pairs of a color density value and a label of the true valueimage for each pixel in a single color image of each color, R, G, or Bshown in FIG. 2 is the training data.

The learning of the neural network means adjusting the weight w suchthat, with respect to any input/output pair (x_(n),d_(n)), the outputy(x_(n):w) of the neural network when given an input x_(n), becomesclose to the output d_(n) as much as possible. An error function is ascale for measuring the closenessy(x _(n) :w)≈d _(n)  [Math. 8]between the training data and the function expressed by use of theneural network. The error function is also called a loss function. Anerror function E(w) used in the image analysis method according to theembodiment is expressed by Formula 6 below. Formula 6 is called crossentropy.[Math. 9]E(w)=−Σ_(n=1) ^(N)Σ_(k=1) ^(K) d _(nk) log y _(k)(x _(n) ;w)  (Formula6)

A method for calculating the cross entropy in Formula 6 is described. Inthe output layer 50 b of the neural network 50 to be used in the imageanalysis method according to the embodiment, that is, in the last layerof the neural network, an activation function is used that classifiesinputs x into a finite number of classes according to the contents. Theactivation function is called a softmax function and expressed byFormula 7 below. It is assumed that, in the output layer 50 b, the nodesare arranged by the same number as the number of classes k. It isassumed that the total input u of each node k (k=1, . . . K) in theoutput layer L is given as u_(k)(L) from the outputs of the previouslayer L−1. Accordingly, the output of the k-th node in the output layeris expressed by Formula 7 below.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 10} \right\rbrack & \; \\{{y_{k} \equiv z_{k}^{(L)}} = \frac{\exp\mspace{11mu}\left( u_{k}^{(L)} \right)}{{\sum^{K}}_{j = 1}{\exp\mspace{11mu}\left( u_{k}^{(L)} \right)}}} & \left( {{Formula}\mspace{14mu} 7} \right)\end{matrix}$

Formula 7 is the softmax function. The sum of outputs y₁, . . . , y_(K)determined by Formula 7 is always 1.

When each class is expressed as C₁, . . . , C_(K), output y_(K) of nodek in the output layer L (that is, u_(k) ^((L))) represents theprobability that a given input x belongs to class C_(K). Refer toFormula 8 below. The input x is classified into a class which allows theprobability expressed by Formula 8 to be the largest.[Math. 11]p(C _(k) |x)=y _(k) =z _(k) ^((L))  (Formula 8)

In learning of the neural network, a function expressed by the neuralnetwork is considered as a model of the posterior probability of eachclass, the likelihood of weights w to the training data is evaluatedunder such a probabilistic model, and weights w that maximize thelikelihood are selected.

It is assumed that target output d_(n) by the softmax function ofFormula 7 is 1 only if the output is a correct class, and otherwise,target output d_(n) is 0. In a case where the target output is expressedin a vector format of d_(n)=[d_(n1), . . . , d_(nK)], if, for example,the correct class of input x_(n) is C₃, only target output d_(n3) is 1,and the other target outputs are 0. When coding is performed in thismanner, the posterior distribution is expressed by Formula 9 below.[Math. 12]p(d|x)=Π_(k=1) ^(K) p(C _(k) |x)^(d) ^(k)   (Formula 9)

Likelihood L(w) of weights w to the training data {(x_(n),d_(n))}(n=1, .. . , N) is expressed by Formula 10 below. When the logarithm oflikelihood L(w) is taken and the sign is inverted, the error function ofFormula 6 is derived.[Math. 13]L(w)=Π_(n=1) ^(N) p(d _(n) |x _(n) ;w)=Π_(n=1) ^(N)Π_(k=1) ^(K) p(C _(k)|x _(n))^(d) ^(nk) =Π_(n+1) ^(N)Π_(k=1) ^(K)(y _(k)(x;w))^(d) ^(nk)  (Formula 10)

Learning means minimizing error function E(w) calculated on the basis ofthe training data with respect to parameter w of the neural network. Inthe image analysis method according to the embodiment, error functionE(w) is expressed by Formula 6.

Minimizing error function E(w) with respect to parameter w has the samemeaning as finding a local minimum point of function E(w). Parameter wis a weight of the connection between nodes. A minimum point of weight wis obtained by iterative calculation of iteratively updating parameter wfrom an arbitrary initial value as a starting point. An example of suchcalculation is the gradient descent method.

In the gradient descent method, a vector expressed by Formula 11 belowis used.

$\begin{matrix}\left\lbrack {{Math}.\mspace{11mu} 14} \right\rbrack & \; \\{{\nabla\; E} = {\frac{\partial E}{\partial w} = \left\lbrack {\frac{\partial E}{\partial w_{1}},\ldots\;,\frac{\partial E}{\partial w_{M}}} \right\rbrack^{T}}} & \left( {{Formula}\mspace{14mu} 11} \right)\end{matrix}$

In the gradient descent method, processing to move the value of currentparameter w in the negative gradient direction (that is, −∇E) isiterated many times. If it is assumed that w^((t)) is the current weightand that w^((t+1)) is the weight after moving, the calculation accordingto the gradient descent method is expressed by Formula 12 below. Value tmeans the number of times the parameter w is moved.

[Math.  15] $\begin{matrix}{{w^{({t + 1})} = {w^{(t)} - {\epsilon{\nabla{E\left\lbrack {{Math}.\mspace{14mu} 16} \right\rbrack}}}}}\epsilon} & \left( {{Formula}\mspace{14mu} 12} \right)\end{matrix}$

The above symbol is a constant that determines the magnitude of theupdate amount of parameter w, and is called a learning coefficient. Byiterating the calculation expressed by Formula 12, as the value tincreases, error function E(w^((t))) decreases, and parameter w reachesa minimum point.

It should be noted that the calculation according to Formula 12 may beperformed on all the training data (n=1, . . . , N) or may be performedon only part of the training data. The gradient descent method that isperformed on only part of the training data is called a stochasticgradient descent method. In the image analysis method according to theembodiment, the stochastic gradient descent method is used.

(Image Analysis Process)

FIG. 11 shows a function block diagram of the image analysis apparatus200A, which performs an image analysis process of generating the data 83regarding cell morphology from the analysis target image 78. Aprocessing unit 20A of the image analysis apparatus 200A includes ananalysis data generation unit 201, an analysis data input unit 202, ananalysis unit 203, and a cell nucleus area detection unit 204. Thesefunction blocks are realized when a program according to the presentdisclosure for causing a computer to execute the image analysis processis installed in the storage unit 23 or the memory 22 of the processingunit 20A, and the program is executed by the CPU 21. The training datadatabase (DB) 104 and the algorithm database (DB) 105 are provided fromthe deep learning apparatus 100A through the storage medium 98 or thenetwork 99, and are stored in the storage unit 23 or the memory 22 ofthe processing unit 20A.

Each analysis target image 78 is captured by the imaging apparatus 400and is stored in the storage unit 23 or the memory 22 of the processingunit 20A. The first deep learning algorithm 60 and the second deeplearning algorithm 61 which have been trained and which includeconnection weights w are stored in the algorithm database 105, inassociation with, for example, the morphological-classification-basedcell type or cell feature to which the analysis target cell belongs. Thefirst deep learning algorithm 60 and the second deep learning algorithm61 function as program modules which are part of the program that causesthe computer to execute the image analysis process. That is, the firstdeep learning algorithm 60 and the second deep learning algorithm 61 areused by the computer including a CPU and a memory. The first deeplearning algorithm 60 and the second deep learning algorithm 61 are usedin order to identify which of the morphologically classified types of aplurality of cells belonging to a predetermined cell group correspondsto the analysis target cell, and in order to generate the data 83regarding the cell morphology. The generated data is outputted asnecessary. The CPU 21 of the processing unit 20A causes the computer tofunction so as to execute specific information calculation or processingaccording to the use objective. Specifically, the CPU 21 of theprocessing unit 20A generates the data 83 regarding cell morphology, byuse of the first deep learning algorithm 60 and the second deep learningalgorithm 61 stored in the storage unit 23 or the memory 22. The CPU 21of the processing unit 20A inputs the analysis data 81 to the inputlayer 60 a and outputs, from the output layer 60 b, the feature quantityof the analysis image 78 calculated by the first deep learning algorithm60. The CPU 21 of the processing unit 20A inputs the feature quantityoutputted from the first deep learning algorithm 60, into the inputlayer 61 a of the second deep learning algorithm, and outputs, from theoutput layer 61 b, a label value corresponding to themorphological-classification-based cell type or cell feature to whichthe analysis target cell has been identified as belonging. Withreference to the function blocks shown in FIG. 11 , the processes ofsteps S21 and S22 are performed by the analysis data generation unit201. The processes of steps S23, S24, S25, and S27 are performed by theanalysis data input unit 202. The process of step S26 is performed bythe analysis unit 203.

With reference to FIG. 12 , description is given of an example of theimage analysis process of generating the data 83 regarding the cellmorphology from the analysis target image 78 performed by the processingunit 20A.

First, the processing unit 20A obtains analysis images 78. Each analysisimage 78 is obtained via the I/F unit 25 through an operation by a user,from the imaging apparatus 400, from the storage medium 98, or via anetwork.

In step S21, similar to the step S11 shown in FIG. 9 , the obtainedanalysis image 78 is converted into brightness Y, first hue Cb, andsecond hue Cr, and the tone vector data 80 is generated in accordancewith the procedure described in the analysis data generation methodabove.

Next, in step S22, the processing unit 20A generates the analysis data81 from the tone vector data 80 in accordance with the proceduredescribed in the analysis data generation method above.

Next, in step S23, the processing unit 20A obtains the first deeplearning algorithm and the second deep learning algorithm stored in thealgorithm database 105.

Next, in step S24, the processing unit 20A inputs the analysis data 81to the first deep learning algorithm. In accordance with the proceduredescribed in the image analysis method above, the processing unit 20Ainputs the feature quantity outputted from the first deep learningalgorithm to the second deep learning algorithm. Then, a label valuecorresponding to the cell type or cell feature to which the analysistarget cell included in the analysis image is determined as belonging isoutputted from the second deep learning algorithm. The processing unit20A stores this label value into the memory 22 or the storage unit 23.

In step S27, the processing unit 20A determines whether identificationhas been performed with respect to all the analysis images 78 initiallyobtained. When identification with respect to all the analysis images 78has ended (YES), the processing unit 20A advances to step S26, andoutputs an analysis result including the data 83 regarding the cellmorphology. When identification with respect to all the analysis images78 has not ended (NO), the processing unit 20A advances to step S25, andperforms the processes of steps S21 to step S25 with respect to theanalysis images 78 for which the identification has not been performed.

According to the present embodiment, identification of cell type andcell feature based on morphological classification can be performedregardless of the skill of the examiner, and morphology examinations canbe suppressed from varying.

<Computer Program>

The present disclosure includes a computer program for performing imageanalysis for analyzing cell morphology, the computer program configuredto cause a computer to execute the processes of steps S11 to S17 and/orS21 to S27.

Further, an embodiment of the present disclosure relates to a programproduct such as a storage medium having stored therein the computerprogram. That is, the computer program is stored in a storage mediumsuch as a hard disk, a semiconductor memory device such as a flashmemory or an optical disk. The storage form of the program into thestorage medium is not restricted as long as the above-presentedapparatus can read the program. The storage in the storage medium ispreferably performed in a nonvolatile manner.

3. Image Analysis System 2

<Configuration of Image Analysis System 2>

Another mode of the image analysis system is described.

FIG. 13 shows a configuration example of a second image analysis system.The second image analysis system includes the user-side apparatus 200,and the user-side apparatus 200 operates as an integrated-type imageanalysis apparatus 200B. The image analysis apparatus 200B isimplemented as a general purpose computer, for example, and performsboth the deep learning process and the image analysis process describedwith respect to the image analysis system 1 above. That is, the secondimage analysis system is a stand-alone-type system that performs deeplearning and image analysis on the user side. In the second imageanalysis system, the integrated-type image analysis apparatus 200Binstalled on the user side performs both functions of the deep learningapparatus 100A and the image analysis apparatus 200A according to thefirst embodiment.

In FIG. 13 , the image analysis apparatus 200B is connected to theimaging apparatus 400. The imaging apparatus 400 captures trainingimages 70 during the deep learning process, and captures analysis targetimages 78 during the image analysis process.

<Hardware Configuration>

The hardware configuration of the image analysis apparatus 200B issimilar to the hardware configuration of the user-side apparatus 200shown in FIG. 7 .

<Function Block and Processing Procedure>

FIG. 14 shows a function block diagram of the image analysis apparatus200B. A processing unit 20B of the image analysis apparatus 200Bincludes the training data generation unit 101, the training data inputunit 102, the algorithm update unit 103, the analysis data generationunit 201, the analysis data input unit 202, the analysis unit 203, and acell nucleus detection unit 204. These function blocks are realized whena program for causing a computer to execute the deep learning processand the image analysis process is installed in the storage unit 23 orthe memory 22 of the processing unit 20B, and the program is executed bythe CPU 21. The training data database (DB) 104 and the algorithmdatabase (DB) 105 are stored in the storage unit 23 or the memory 22 ofthe processing unit 20B, and both are used in common during the deeplearning and the image analysis process. The first neural network 60 andthe second neural network 61 that have been trained are stored inadvance in the algorithm database 105 in association with themorphological-classification-based cell type or cell feature to whichthe analysis target cell belongs, for example. With connection weights wupdated by the deep learning process, the deep learning algorithm 60 isstored in the algorithm database 105. It is assumed that each trainingimage 70 is captured in advance by the imaging apparatus 400 and isstored in advance in the training data database (DB) 104 or in thestorage unit 23 or the memory 22 of the processing unit 20B. It is alsoassumed that each analysis target image 78 of the analysis targetpreparation is captured in advance by the imaging apparatus 400, and isstored in advance in the storage unit 23 or the memory 22 of theprocessing unit 20B.

The processing unit 20B of the image analysis apparatus 200B performsthe process shown in FIG. 9 during the deep learning process, andperforms the process shown in FIG. 12 during the image analysis process.With reference to the function blocks shown in FIG. 14 , during the deeplearning process, the processes of steps S11, S12 S16, and S17 areperformed by the training data generation unit 101. The process of stepS13 is performed by the training data input unit 102. The processes ofsteps S14 and S15 are performed by the algorithm update unit 103. Duringthe image analysis process, the processes of steps S21 and S22 areperformed by the analysis data generation unit 201. The processes ofsteps S23, S24, S25, and S27 are performed by the analysis data inputunit 202. The process of step S26 is performed by the analysis unit 203.

The procedure of the deep learning process and the procedure of theimage analysis process performed by the image analysis apparatus 200Bare similar to the procedures respectively performed by the deeplearning apparatus 100A and the image analysis apparatus 200A. However,the image analysis apparatus 200B obtains the training image 70 from theimaging apparatus 400.

In the image analysis apparatus 200B, the user can confirm theidentification accuracy of the classifier. If the identification resultby the classifier is different from the identification result obtainedthrough image observation by the user, the first deep learning algorithmand the second deep learning algorithm can be re-trained by using theanalysis data 81 as training data 78 and by using, as the label value77, the identification result obtained through image observation by theuser. Accordingly, the training efficiency of the first neural network50 and the second neural network 51 can be improved.

3. Image Analysis System 3

<Configuration of Image Analysis System 3>

Another mode of the image analysis system is described.

FIG. 15 shows a configuration example of a third image analysis system.The third image analysis system includes the vendor-side apparatus 100and the user-side apparatus 200. The vendor-side apparatus 100 operatesas an integrated-type image analysis apparatus 100B, and the user-sideapparatus 200 operates as a terminal apparatus 200C. The image analysisapparatus 100B is implemented as a general purpose computer, forexample, and is a cloud-server-side apparatus which performs both thedeep learning process and the image analysis process described withrespect to the image analysis system 1. The terminal apparatus 200C isimplemented as a general purpose computer, for example, and is auser-side terminal apparatus which transmits images of the analysistarget to the image analysis apparatus 100B through the network 99, andreceives analysis result images from the image analysis apparatus 100Bthrough the network 99.

In the third image analysis system, the integrated-type image analysisapparatus 100B installed on the vendor side performs both functions ofthe deep learning apparatus 100A and the image analysis apparatus 200A.Meanwhile, the third image analysis system includes the terminalapparatus 200C, and provides the terminal apparatus 200C on the userside with an input interface for the analysis image 78 and an outputinterface for the analysis result image. That is, the third imageanalysis system is a cloud-service-type system in which the vendor side,which performs the deep learning process and the image analysis process,provides an input interface for providing the analysis image 78 to theuser side, and an output interface for providing the data 83 regardingcell morphology to the user side. The input interface and the outputinterface may be integrated.

The image analysis apparatus 100B is connected to the imaging apparatus300, and obtains the training image 70 captured by the imaging apparatus300.

The terminal apparatus 200C is connected to the imaging apparatus 400,and obtains the analysis target image 78 captured by the imagingapparatus 400.

<Hardware Configuration>

The hardware configuration of the image analysis apparatus 100B issimilar to the hardware configuration of the vendor-side apparatus 100shown in FIG. 6 . The hardware configuration of the terminal apparatus200C is similar to the hardware configuration of the user-side apparatus200 shown in FIG. 7 .

<Function Block and Processing Procedure>

FIG. 16 shows a function block diagram of the image analysis apparatus100B. A processing unit 10B of the image analysis apparatus 100Bincludes the training data generation unit 101, the training data inputunit 102, the algorithm update unit 103, the analysis data generationunit 201, the analysis data input unit 202, the analysis unit 203, andthe cell nucleus area detection unit 204. These function blocks arerealized when a program for causing a computer to execute the deeplearning process and the image analysis process is installed in thestorage unit 13 or the memory 12 of the processing unit 10B, and theprogram is executed by the CPU 11. The training data database (DB) 104and the algorithm database (DB) 105 are stored in the storage unit 13 orthe memory 12 of the processing unit 10B, and both are used in commonduring the deep learning and the image analysis process. The firstneural network 50 and the second neural network 51 are stored in advancein the algorithm database 105 in association with themorphological-classification-based cell type or cell feature to whichthe analysis target cell belongs, for example, and are stored in thealgorithm database 105 as the first deep learning algorithm 60 and thesecond deep learning algorithm 61, with connection weights w updated bythe deep learning process.

Each training image 70 is captured in advance by the imaging apparatus300 and is stored in advance in the training data database (DB) 104 orin the storage unit 13 or the memory 12 of the processing unit 10B. Itis assumed that each analysis target image 78 is captured by the imagingapparatus 400 and is stored in advance in the storage unit 23 or thememory 22 of the processing unit 20C of the terminal apparatus 200C.

The processing unit 10B of the image analysis apparatus 100B performsthe process shown in FIG. 9 during the deep learning process, andperforms the process shown in FIG. 12 during the image analysis process.With reference to the function blocks shown in FIG. 16 , the processesof steps S11, S12, S16, and S17 are performed by the training datageneration unit 101 during the deep learning process. The process ofstep S13 is performed by the training data input unit 102. The processesof steps S14 and S15 are performed by the algorithm update unit 103.During the image analysis process, the processes of steps S21 and S22are performed by the analysis data generation unit 201. The processes ofsteps S23, S24, S25, and S27 are performed by the analysis data inputunit 202. The process of step S26 is performed by the analysis unit 203.

The procedure of the deep learning process and the procedure of theimage analysis process performed by the image analysis apparatus 100Bare similar to the procedures respectively performed by the deeplearning apparatus 100A and the image analysis apparatus 200A accordingto the first embodiment.

The processing unit 10B receives the analysis target image 78 from theterminal apparatus 200C on the user side, and generates the trainingdata 75 in accordance with steps S11 to S17 shown in FIG. 9 .

In step S26 shown in FIG. 12 , the processing unit 10B transmits theanalysis result including the data 83 regarding cell morphology, to theterminal apparatus 200C on the user side. In the terminal apparatus 200Con the user side, the processing unit 20C outputs the received analysisresult to the output unit 27.

In this manner, by transmitting the analysis target image 78 to theimage analysis apparatus 100B, the user of the terminal apparatus 200Ccan obtain the data 83 regarding cell morphology as the analysis result.

According to the image analysis apparatus 100B of the third embodiment,the user can use the classifier, without obtaining the training datadatabase 104 and the algorithm database 105 from the deep learningapparatus 100A. Accordingly, the service for identifying the cell typeand cell feature based on morphological classification can be providedas a cloud service.

4. Other Embodiments

The outlines and specific embodiments of the present disclosure havebeen described. However, the present disclosure is not limited to theoutlines and embodiments described above.

In the present disclosure, an example of a method for generating thetraining data 75 by converting the tone into brightness Y, first hue Cb,and second hue Cr has been described. However, the conversion of thetone is not limited thereto. Without converting the tone, the threeprimary colors of red (R), green (G), and blue (B), for example, may bedirectly used. Alternatively, two primary colors obtained by excludingone hue from the primary colors may be used. Alternatively, one primarycolor (for example, green (G) only) obtained by selecting any one of thethree primary colors of red (R), green (G), and blue (B) may be used.The conversion into three primary colors of cyan (C), magenta (M), andyellow (Y) may be employed. Also, for example, the analysis target image78 is not limited to a color image of the three primary colors of red(R), green (G), and blue (B), and may be a color image of two primarycolors. It is sufficient that the image includes one or more primarycolors.

In the training data generation method and the analysis data generationmethod described above, in step S11, the processing unit 10A, 20B, 10Bgenerates the tone matrix 72 y, 72 cb, 72 cr from the training image 70.However, the training image 70 may be the one converted into brightnessY, first hue Cb, and second hue Cr. That is, the processing unit 10A,20B, 10B may originally obtain brightness Y, first hue Cb, and secondhue Cr, directly from the virtual slide scanner or the like, forexample. Similarly, in step S21, although the processing unit 20A, 20B,10B generates the tone matrix 72 y, 72 cb, 72 cr from the analysistarget image 78, the processing unit 20A, 20B, 10B may originally obtainbrightness Y, first hue Cb, and second hue Cr, directly from the virtualslide scanner or the like, for example.

Other than RGB and CMY, various types of color spaces such as YUV andCIE L*a*b* can be used in image obtainment and tone conversion.

In the tone vector data 74 and the tone vector data 80, for each pixel,information of tone is stored in the order of brightness Y, first hueCb, and second hue Cr. However, the order of storing the information oftone and the handling order thereof are not limited thereto. However,the arrangement order of the information of tone in the tone vector data74 and the arrangement order of the information of tone in the tonevector data 80 are preferably the same with each other.

In each image analysis system, the processing unit 10A, 10B is realizedas an integrated apparatus. However, the processing unit 10A, 10B maynot necessarily be an integrated apparatus. Instead, a configuration maybe employed in which the CPU 11, the memory 12, the storage unit 13, theGPU19, and the like, are arranged at separate places; and these areconnected through a network. Also, the processing unit 10A, 10B, theinput unit 16, and the output unit 17 may not necessarily be disposed atone place, and may be respectively arranged at separate places andcommunicably connected with one another through a network. This alsoapplies to the processing unit 20A, 20B, 20C.

In the first to third embodiments, the function blocks of the trainingdata generation unit 101, the training data input unit 102, thealgorithm update unit 103, the analysis data generation unit 201, theanalysis data input unit 202, and the analysis unit 203 are executed bya single CPU 11 or a single CPU 21. However, these function blocks maynot necessarily be executed by a single CPU, and may be executed in adistributed manner by a plurality of CPUs. These function blocks may beexecuted in a distributed manner by a plurality of GPUs, or may beexecuted in a distributed manner by a plurality of CPUs and a pluralityof GPUs.

In the second and third embodiments, the program for performing theprocess of each step described with reference to FIG. 9, 12 is stored inadvance in the storage unit 13, 23. Instead, the program may beinstalled in the processing unit 10B, 20B from a computer-readable,non-transitory, and tangible storage medium 98 such as a DVD-ROM or aUSB memory, for example. Alternatively, the processing unit 10B, 20B maybe connected to the network 99 and the program may be downloaded from,for example, an external server (not shown) through the network 99 andinstalled.

In each image analysis system, the input unit 16, 26 is an input devicesuch as a keyboard or a mouse, and the output unit 17, 27 is realized asa display device such as a liquid crystal display. Instead, the inputunit 16, 26, and the output unit 17, 27 may be integrated to realize atouch-panel-type display device. Alternatively, the output unit 17, 27may be implemented by a printer or the like.

In each image analysis system described above, the imaging apparatus 300is directly connected to the deep learning apparatus 100A or the imageanalysis apparatus 10B. However, the imaging apparatus 300 may beconnected to the deep learning apparatus 100A or the image analysisapparatus 100B via the network 99. Also with respect to the imagingapparatus 400, similarly, although the imaging apparatus 400 is directlyconnected to the image analysis apparatus 200A or the image analysisapparatus 200B, the imaging apparatus 400 may be connected to the imageanalysis apparatus 200A or the image analysis apparatus 200B via thenetwork 99.

5. Effect of Deep Learning Algorithm

In order to validate the effect of the deep learning algorithm, the cellidentification accuracy by a cell identification method usingconventional machine learning was compared with the cell identificationaccuracy by the cell identification method using the deep learningalgorithm of the present disclosure.

A peripheral blood smear preparation was created by a smear preparationcreation apparatus SP-1000i, and cell image capturing was performed by ahemogram automatic analyzer DI-60. May-Giemsa stain was used as thestain.

Cell identification by the conventional machine learning was performedby the hemogram automatic analyzer DI-60. Three persons including adoctor and an experienced laboratory technician observed the image toperform the validation.

FIG. 17 shows the comparison result of hemocyte classification accuracy.When the deep learning algorithm was used, discrimination at a higheraccuracy than in the conventional method was achieved.

Next, it was examined whether the deep learning algorithm of the presentdisclosure was able to identify morphological features observed inmyelodysplastic syndromes (MDS). FIG. 18 shows the result.

As shown in FIG. 18 , morphological nucleus abnormality, vacuolation,granule distribution abnormality, and the like were accuratelyidentified.

From the above result, it was considered that the deep learningalgorithm of the present disclosure can accurately identify the type ofcell and the feature of cell based on morphological classification.

What is claimed is:
 1. An image analysis method comprising: inputtinganalysis data into a deep learning algorithm having a neural networkstructure, the analysis data being generated from an image of ananalysis target blood cell and including information regarding theanalysis target blood cell; and generating, by use of the deep learningalgorithm, a probability that the analysis target blood cell belongs toeach of morphology classifications of blood cells, wherein the analysisdata comprises vector data for each of pixels in the image of theanalysis target blood cell.
 2. The image analysis method of claim 1,wherein the morphology classification indicates a type of blood cell. 3.The image analysis method of claim 2, wherein the type of blood cellincludes at least one of segmented neutrophil, band neutrophil,metamyelocyte, bone marrow cell, blast, lymphocyte, atypical lymphocyte,monocyte, eosinophil, basophil, erythroblast, giant platelet, plateletaggregate, or megakaryocyte.
 4. The image analysis method of claim 2,further comprising outputting the type of blood cell based on theprobability generated by the deep learning algorithm.
 5. The imageanalysis method of claim 1, wherein the morphology classificationindicates a type of abnormal finding.
 6. The image analysis method ofclaim 5, wherein the type of abnormal finding includes at least one ofmorphological nucleus abnormality, presence of vacuole, granulemorphological abnormality, granule distribution abnormality, presence ofabnormal granule, cell size abnormality, presence of inclusion body, orbare nucleus.
 7. The image analysis method of claim 5, furthercomprising outputting the type of abnormal finding based on theprobability generated by the deep learning algorithm.
 8. The imageanalysis method of claim 1, wherein the morphology classificationsindicate a type of blood cell and a type of abnormal finding; andoutputting the type of blood cell and the type of abnormal finding basedon the probability generated by the deep learning algorithm.
 9. Theimage analysis method of claim 1, wherein the morphology classificationsindicate a type of blood cell and a type of abnormal finding; and theinputting analysis data comprises inputting the analysis data into afirst deep learning algorithm having a neural network structure forclassification of the type of the blood cell and inputting the analysisdata into a second deep learning algorithm having a neural networkstructure and being different from the first deep learning algorithm forclassification of the abnormal finding.
 10. The image analysis method ofclaim 1, further comprising trimming the image of an analysis targetblood cell so that some other cells other than the target blood cell areremoved from the image before inputting the analysis data into the deeplearning algorithm.
 11. The image analysis method of claim 1, whereinthe analysis target cell in the image has been stained.
 12. The imageanalysis method of claim 11, wherein the analysis target cell has beenstained by Wright's staining, Giemsa staining, Wright-Giemsa staining,or May-Giemsa staining.
 13. The image analysis method of claim 1,wherein the analysis target cell is contained in peripheral blood. 14.The image analysis method of claim 1, wherein vector components in thevector data correspond to hues of the pixels.
 15. An image analysisapparatus comprising a processor configured to: inputting analysis datainto a deep learning algorithm having a neural network structure, theanalysis data being generated from an image of an analysis target bloodcell and including information regarding the analysis target blood cell;and generating, by use of the deep learning algorithm, a probabilitythat the analysis target blood cell belongs to each of morphologyclassifications of blood cells, wherein the analysis data comprisesvector data for each of pixels in the image of the analysis target bloodcell.
 16. The image analysis apparatus of claim 15, wherein themorphology classification indicates a type of blood cell.
 17. The imageanalysis apparatus of claim 15, wherein the morphology classificationindicates a type of abnormal finding.
 18. The image analysis apparatusof claim 17, wherein the type of abnormal finding includes at least oneof morphological nucleus abnormality, presence of vacuole, granulemorphological abnormality, granule distribution abnormality, presence ofabnormal granule, cell size abnormality, presence of inclusion body, orbare nucleus.
 19. A non-transitory computer readable medium storingprograms executable by a processor to: inputting analysis data into adeep learning algorithm having a neural network structure, the analysisdata being generated from an image of an analysis target blood cell andincluding information regarding the analysis target blood cell; andgenerating, by use of the deep learning algorithm, a probability thatthe analysis target blood cell belongs to each of morphologyclassifications of blood cells, wherein the analysis data comprisesvector data for each of pixels in the image of the analysis target bloodcell.